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METHODOLOGY BASED ON FACET THEORY (MODIFIED SET THEORY) 
WAS USED IN TEST CONSTRUCTION AND ANALYSIS TO PROVIDE AN 
EFFICIENT TOOL OF EVALUATION FOR VOCATIONAL GUIDANCE AND 
VOCATIONAL SCHOOL USE. THE TYPE OF TEST DEVELOPMENT 
UNDERTAKEN WAS LIMITED TO THE USE OF NONVERBAL PICTORIAL 
ITEMS. ITEMS FOR TESTING ABILITY TO IDENTIFY ELEMENTS 
BELONGING TO AN ORDERED SET (ANALOGY TESTING) WERE 
CONSTRUCTED QN THE BASIS OF FACET DESIGN, AND THE ELEMENTS OF 
THE SET WERE PRESENTED IN SIMPLE DIAGRAMMATIC DRAWINGS. THE 
TWO TESTS DEVISED IN THIS PROJECT WERE AN ANALYTICAL ABILITY 
TEST AND A MECHANICAL COMPREHENSION TEST. THE ATTRIBUTES, 
FUNCTIONS, AND ARRANGEMENTS EMPLOYED IN THE ANALYTICAL 
ABILITY TEST WERE DESCRIBED. THE ITEMS EMPLOYED IN THE 
ANALYTICAL ABILITY TEST WERE OF THREE CLASSES «*) CLASS A - 
WHERE FIGURES VARIED IN SIZE, DIRECTION, OR PLACE, (2) CLASS 
B - WHERE SHAPE WAS AN ATTRIBUTE WHICH VARIED, AND (3) CLASS 
C - WHERE FUNCTION WAS EXHIBITED BY DIFFERENT ATTRIBUTES OR 
DIFFERENT SETS OF VALUES OF AN ATTRIBUTE. DESCRIPTIONS WERE 
INCLUDED OF (t) DISTRACTORS OF THE ANALYTICAL ABILITY TEST, 

(2) ANALYTICAL ABILITY TEST BOOKLETS, (3) ITEMS OF THE 
MECHANICAL COMPREHENSION TEST, (4) MECHANICAL COMPREHENSION 
TEST BOOKLETS, AND (5) SOME NEW METHODS OF ITEM ANALYSIS. 

DATA COLLECTION INCLUDED THE ADMINISTRATION OF THE 
CONSTRUCTED TESTS TO A SAMPLE OF 637 PUPILS IN GRADES 7, 0, 
AND 9. CONCLUSIONS WERE (1) FACET DESIGN COULD BE EMPLOYED IN 
ITEM CONSTRUCTION, (2) THE DEGREE OF SUCCESS (TEST 
DEVELOPMENT) ACHIEVED WAS UNEVEN, DIFFERING WITH THE KIND OF 
TEST AND THE NATURE OF ANALYSIS CONDUCTED, AND (3) VARIOUS 
REGIONS OF CLASSES, ARRANGEMENTS, AND FUNCTIONS, AS WELL AS 
THE SUB TESTS OF THE MECHANICAL TEST SHOULD PROVE IMPORTANT 
FOR PREDICTIVE MEASUREMENT. (R S) 
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1 # OBJECTIVES AND RATIONALE OF THE S1OT 



1.1* General Objectives 



The present project constitutes a first attempt at implem e nting 
a new methodology of test construction and analysis, which Is based 
on face theory* This theory has been developed by the principal 
investigator, who has also shown the fruitfulness of the facet approach 
for the reinterpretation of previous research on intellectual abilities 
(see *: Terences 1*, 6, 8, 9$ 11) « However, apart from a few minor 
attempts (as yet unreported) this is the first time a methodology 
based on facet theory is systematically used in test development* 



Development of this methodology was stimulated in part by the 
need for further progrtvs in the field of mental testing* Little 
i mpro vement has been made over the validity ceilings of standard 
tests attained by Spearman and Binet or In the original Alpha tests 
of the Amsrloan army* In most cases, the validity ceilings do not 
rise above *5 or *6* It would seem that in order to arrive at 






'VV -^3 J ’ 






substantially higher validities, a new approach in test construction 



would be necessary* As will be shown. 
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approach taken in the 
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present study, holds promise of eventually bringing about improvemmat 
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i»Mdiate practical preble* leading to this project, wae a; 
the need for an efficient tool of evaluation for the purposes of 
vocational guidance and school use* Although it mqr be assumed that 
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Section 1*1 (2) 



both general intelligence and mechanical ability are needed for success 
in vocational training* there is as yet insufficient information on the 
relationship existing between the different kinds of abilities in this 

area* 

Tests developed in this project differentiate more sharply 
between general intelligence and mechanical aptitude* than do previous 
tests* Thus* decisions as to whether a particular student is better 
qualified for academic or for vocational training can be made more 
accurate* as can differential placement within vocational areas* Analyses 
of the interrelationships between different abilities that have been 
carried out in the present study represent a first step towards attaining 
this objective. 

In view of the varied degrees of literacy in the populations in 
which the tests might be used* we restricted ourselves to pictorial 
tests* Two types of tests were developed in this projects (l) Ana- 
lytical^ ability tests | *' » these employed abstract figures and were com- 
pletely nonverbal. (2) A mechanical comprehension test in which each 
item presented e situation pictorially about which a question was asked 
verbally* Various analyses were carried out on the data obtainad with 



1$ ■K-'. 

f -.4 .'4i- ■ ' 



^■.t“ 












these taste* In some of these* new methods of analysis recently developed 
by the principal investigator were employed* As will be seen in the 
following* the results attest to ths potential f ruitfulnsss of tha f scat 
approach in predicting both the degres of difficulty and the structure 
of the interrelationship between varieue tests and subteste* ^ 
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1*2* Rationale of Item Construction 



The approach to mental ability tests which is taken in the present 
project has been described previously by the principal investigator 
(see references U> 6, 3, ?)• It was first developed during weekly 
meetings held during the course of several years with member* of the 
Israel Amy Psychotechnical unit, where a new type of definition of 
intelligence, involving analytical ability, was arrived at* This new 
definitional approach aimed, among other things, at ascertaining ’.Jhat 
common system underlies the testing programs conducted in England, 
America and other countries over the past sixty years* A definitional 
system of what all tnese programs have in common was developed* Here 
a "faceted definition 11 was employed (see reference 9)» 
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Given any two sets of elements A and E, which we call facets* 
their Cartesian space is the set of all pairs cf elements ab, where 
a is an element of A and b is an element of B. A Cartesian space 
may consist of any number of facets, or sets of elements; with n 
facets, any one point in the Cartesian space has n component elements* 

A set of abilities of a given kind is defined simultaneously 
by a "faceted definition" which makes use of two or more facets, in 

such a wpy that any element of the Cartesian space defines one variety 

.£& the :h’rs%- WMmthm the &i‘ th* 

of the ability* 
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’ The definition Which ma finally *niwd it la tha following* 
'M An act of a subject is IntelUgmt to the (extent) te» 

( tester) sae (demonstrating) 
a correct perception of an unexhibited logical (aspect) 
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Section 1*2 (2) 



of a (relation) intended by the tester, on the 
basis of another (exhibited) logical (aspect) 
of that relation that is correctly perceived 
by the subject* 



Italics a re used to indicate the constant features of the concept 
of intelligence# while the parentheses indicate facets# which are sets 
of ideas rather than constants* 



The substance of a question in an intelligence test can be 
characterized as exhibiting a logical aspect of a relation* If the 
subject* 3 response is correct# it will demonstrate a correct perception 
of an unexhibited logical aspect of the same relation* Given this 
focus on logical aspects of relations# items which test analytical 
ability and items which test achievement can be distinguished* If 
the name or the selection rule of the .relation is exhibited# the test 
is of achievement* If the rule is to be inferred# the test is of 
analytical ability* 



To illustrate this distinction# consider the following two items: 
(1 j Who was the first president of Israel? 

(2) A dog is to a puppy as a cow is to a * 



In the first question the selection rule of the relation - 
first presidency of a country - is exhibited* The Cartesian space 
is that of all first presidents against all countries* The exhibited 
relation is that each countxy is paired only with its own first pre- 
sident* One element of ihis relation# namely Israel# is also exhibited# 
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and another aspect of the relation is required* It is assumed that 
the subject knows what a '‘president" means, what "countiy" means, and 
what "Israel 11 means* What is being tested is whether or not the sub- 
ject possesses the information* This is therefore a test of achievement. 



contrast, the second item exhibits neither the name nor the culling 



rule of a relation* Instead, it exhibits an ordered pair (dog-puppy) 



and the first element of a further ordered pair* Evidently, the intention 
is to see if the subject can infer a two- faceted Cartesian space, within 
which there is a binary relation, of which the two ordered pairs are 
elements* This item is, therefore, one of analytical ability* 

The way in which the relation is exhibited is an important facet 
of the definition* Any of the senses could be used as a mode of com- 
munication, and each mode may define a particular kind of intelligence, 
as is evidenced by the intercorrelation matrix obtained in previous re- 
search (see references 8,9)* Even within the visual mode, using paper 
and pencil tests, there are different kinds of languages: written speech, 
formalized language such as arithmetic and algebra, and pictorial language. 

In focussing only on pictorial tests, we are essentially holding 
a major facet constant, which may help clarify the relation of the em- 
pirical results to the fleets yhich are systematically varied in our tests. 



The approach towards test construction outlined above allows a 
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more eystematic procedure of item construction (see section 1 * 3 ) • Further- 
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and subtests* In general, the relationship between items within the 
framework of facet design should be expected to have its counterpart 
in the empirically obtained correlation matrix, where the size of the 
correlation is related to similarity of facet profiles* These notions 
will be further elucidated when the method of item construction and the 
results of the analyses are described (see especially sections 2*1 and 
3.1.3). 



1.3* Practical Advantages of the Approach Taken 



Various practical advantages may be expected to accrue from the 
facet theory approach which the present project has tried to implement* 



These may be treated under three headings: 



(a) internally systematic construction of items and distractors; 



(b) enhancement of the possibility of revealing lawful empirical relation- 
ships among items and tests; 

(c) using knowledge of the content and statistical structure for the 
parsimonious utilization of the test in the prediction of external 
"criteria* 

The construction of test items has typically been a more or less 
intuitive process* The Investigator usually includes in a first trial 



of and which* on the face of it* seem to him 



U - 



those items he can 
to belong to the universe of items he desires* The decision as to 
whether or not he has had the" right hunch is made to depend on sub- 
sequent item analysis* Bjjr such a trial-and-error procedure, he is 
finally left with a set of items which he includes in his tests* 
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The fallacy of using such statistical techniques as a criterion 
for content has been pointed out many years ago (see reference 2, 
pp#l8l-l8f>) . By contrast, the facet approach supplies the investigator 
with a tool for constructing items systematically* Intuition still, 
has a part to play# since there is no way of formalizing all aspects 
of the process of arriving at a faceted definition* However, once 
such a definition has been set up, the investigator is in possession 
of the building-stones of the i^eros he needs; the construction of items 
becomes thus a much more simple and straightforward process* No 
fallacious use of statistics is involved* 



The suitability of an item for a given test or subteat is 
decided upon on a priori definitional grounds, instead of by sub- 
sequent statistical item analysis* Analysis of correlations between 
items is employed only to test an empirical hypothesis on the relation 
of the observed statistical structure to the faceted definitional 
design* •• -• J>L ' ‘ : 



Not only the items themselves, but also their distractors can * 
be systematically constructed on the basis of those facets which were 
emplsyed in devising the items* ’ This is an important step in for* 
malising the prooess of items construction, becauso distreotors sri 



clearly an integral part of the Litem* to till! ia auafc the 



An illustration of how distractors are constructed through 
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facet design is given in the present study (section 2*2), where it 
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is also shown that this method lends itself to ths systematic con- 






struotion of distraotors diffaring in degrte of attraction* This 



Section 1*3 (3) 



makes it possible to assign meaningful differential scores to each item, 
instead of merely scoring answers as bei. j either "right 11 or "wrong" as 
is customarily done. By assigning such scores it may become feasible 
to employ tests with a much smaller number of items, yet without loss of 
discriminatory power. 

The formalization of item construction is of special importance 
when parallel forms of a test have to be prepared. Unless a facet 
definition is employed, there is no way of hypothesizing beforehand 
whether items included in the two forms may be of comparable difficulty, 
or, more important, of justifying the assertion that they define the 
same kind of ability. By constructing items as well as di a tractors 
on the basis of a faceted definition, it becomes possible in principle 
to devise truly parallel forms, both in the sense of content and of 
statistical structure. This objective will be attained to the extent 
that the investigator has succeeded in including in his faceted definition 
all the behaviorally relevant aspects of items (see also section 3.1.5)# 

So far, the advantages of the approach taken for the construction 
of items and distractors has been discussed. Once the initial form of 
the test has been constructed in this manner, it may be expected that 
the facet design will open up opportunities for further development of 
the test. Analysis of the test results will show to what extent the 
facet structure of the items is reflected by their intercorrelational 
structure. This may lead to a further sharpening of the facet design 
as well as to the construction of additional types of items with other 
facet profiles. 
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* 



The facet approach should ultimately lead to a statistical mapping 
of the structure of the abilities tested* as wall as to establishing 
their empirical relationship with yarious criteria* To the extent 
that the investigator succeeds in this* it will become possible to 
limit the number and the length of tests employed for purposes of 
prediction by using only those test-itesm which correlate most strate- 
gically with the criterion variable* Furthermore* a successful fscat 
design should make it possible to devise shorter fonas of a test with- 
out thereby appreciably reducing its reliability* 
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2* CONSTRUCTION OP TESTS AND FROOKDURES 



2*1# It e»B of Analytical Ability Tost 



Certain decisions had to be mads regarding the type of teat 
which would be constructed within the framework of the present project* 
One possibility would have been to devise tests encompassing a wide 
range of abilities* Data on the relationships between different 
abilities can be found in the literature, and reanalyses of such results 
have been published by the principal investigator elsewhere (see refer- 
ences 8, 9). Thsse appear to show the fruitfulness of the facet 
approach. Although further work with tests tapping a wide range of 
abilities needs to be done, it appeared desirable in the present 
project to limit ourselves to the area of analytical ability, using 
non-verbal items* It was finally decided to construct items testing 
the ability to identify elements belonging to an ordered eet (so-, 
called analogy teats are an example of this type), where the set 
consists of simple geometric drawings* 

Even within this limited area, an indefinite iiy large number 
of items can be constructed on the basis of a facet design* However, 
should the facet approach lead to the prediction of the structure of 
interrelationships between them, a relatively small sample of itmns 
could be used to represent the universe. If sich structural information 
could result within such a narrow area, thiswould certainly justify 
expecting clear structure among areas which differ* more grossly among 
thmuelves* the Vs&A fe - :% #j.' ' u \& 
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The definitional differences among the items of the narrow area 
we have concentrated upon in this project* are small compared to the 
differences in the design between the tests analysed previously, so that 
lire have weaker expectations of finding a stable statistical structure 
in the empirical data* 

When the type of analytical ability selected for this project 
was explored* it became apparent that it included a large number of 
subtypes* A general framework for this type of test was arrived at* 
which permits the construction of a variety of tests. Hence, an 
additional decision had to be made* as to what kinds of items should 
be included within the present project* In the following sections* the 
tost items which were finally decided upon* will be described* 



The test items were in some ways similar to existing analogy 



teats employing figures; a well-known example are Ravenna Progressive 
Matrices* In the following* the final forms of the test items* 
arrived at after an extensive pilot study* will be described* Examples 



of items are tf-vw* in the appendix* 

• ■ ■’ tX? (s’K . 
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An item of the analytieal ability test consists of a three by 






three matrix of squares in whieh the middle square (l*e«* row two* 
column two) is missing* Each square contains a figure* The task 



of tbs subject is to supply the missing square by choosing one of the 

- ' . ■ ■ ' 1 * "■ ' j' 

alternative answers given*^' In erder to aocomplish this task the 



subject must presumably find the nflbs of construction of the item* 
This rule is exiitt>lif&eA by the squares of the matrix; in simpler 
items . - in the right and left hand oolumsis of the matrix ind/or in 
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the upper and lower row of the matrix. In more complex items v the rule 
is exemplified in a different way, as will be explained below. In this 
manner 11 a logical aspect of a relation 11 is exhibited (see the definition 
above, section 1.2). The subject shows his understanding of the rule 
by selecting the correct square from the given alternatives. 



The mle of construction gives the arrangement of three sets 
of three figures in a three by three matrix. In each set, the values 
of the attributes of the three figures and the way in which these values 
are ordered within each set are described by functions « 



Using example 1 of the appendix ae an illustration, it can be 
seen that place (within the email square) and else are varied. Sise 
varies in the orders small, large, medium; that is, by function b* 
(see below). Place varies in the orders upper left comer, upper 



left center, upper left corner; this order is termed function a£ 

(see below)* 

The arrangement in example 1 is by rows and columns; that is, 
function b» orders site by columns and function a* orders place by rows* 

la the following, the attributes, functions and arr an ge me nts 

4 f{\,. . J" | 

employed in the analytical ability test are described* 



Attributes 




There ere two types of attributes s 

Attributes ifooseval^^ be ordered on a continuum 



* ** tiae, number, etc. 

XI* Attributes voich cannot be thus arranged, such as shape* 
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Inspection of the examples in the appendix thows that figures 
of an item in this final version of the test vary in two or more of the 
following attributes: 

T|ype I: Size f direction, and place (within the small square)* 

type II: Shape (o*g* example 2 in the appendix) 



Functions 

In items where the figures varied in attributes of typo I* 
order of the values of this attribute are given by the function* 
Inspection of example 1 shows that in the top and bottom rows figures 
are arranged by size in the following order from left to rights small, 
large, medium* The subject is made to understand by means of the 
instructions that the middle row should exhibit the same characteristics 
as the top and bottom rows, and that the middle figure should be largest 
in size* 



Within each item an attribute may have either two or three values* 
The two-value functions are either a or a* , and the three-value functions - 
b or b» *_,. These are defined as follows: 
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bs f , g, h I where h > g > f 






be* f, h, g j 

iCL^h tv* • ! ?5& 

It will be noted that a* and be are pemutations of a and b 
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Examples of these functions are: 



a - See example 3 in the appendix. The two different values 
for direction are arranged in the columns in the order 
f, f , g, as given by function a. 
ait- . See example !• There are two different values for the 
attribute ‘place 1 - upper left comer and left center « 
arranged in the rows in the order f, g, f, as given by 
function a*. 

b » See example U» Three different values for direction 

(north, northeast, east) are arranged from left to right 
in the order f , g, to, as given toy function b. 
bit- «• See example li* There are three different values of size, 
arranged in the columns in the order: small, large, medium, 
or f, to, g, as given by function b*. 



The above applies only to type I attributes » Type II attributes, 

such as shape, cannot be said to vary by functions, since they cannot 
be ordered* • - •' "• ' a 5 ' 



In the research leading to the construction of these tests, 

’.E>t Vf *»*» Jb i-Hour. 'o* h-. far *.!*:* tsmee, %>• 

other functions were also experimented with* It appeared, however, 

bai'sfe "■■U+j-z df das I * * 

that it was impractical to employ them in a three by three matrix^ 

t§jh Ma */' ^ ©iiaaiiaa dud 

since such a matrix would net allow for an exhibition of the function, 

idsa !»'. .%©■' ts£ 'tk&tM. <&fdter v-a guna 

which would enable the testae to find an unaqulvooal solution. 

EH© id &. i>- u 



dEln an earlier version of the test, an additional function was 

j -i ■ .. - . .. ■ • " ; 

amployed: f, g, h(f,g) - i.e. p the third value, h, was a combination 
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of f and g* This involved drawing a compound figure $ for instance, 
if f and g are a small and a large triangle respectively, h might be a 
small triangle enclosed within or alongside a largo triangle* However, 
when working with such items, it was found that the subject did not pay 
attention to the relationship defined by the function, but rather to 
the number of figures,or else, he saw in the compound figure a new and 
different shape* The use of this function was henceforward dispensed 
with* This experience serves to elucidate an important point regarding 
the construction of items, namely, that aspects of an item attended to 
by the subject may not be those defined by the facet design as intended 
by the researcher* Only careful pretesting or an analysis of the data 
can show whether this is the case* This point will be taken up again 
in the discussion of the results below (section 3*1*6)* 

Arrangements 

Each function orders the values of a set of three figures* There 
are three such sets in a three by three matrix, and there are different 
ways in which these sets can bo arranged in the matrix - by rows, by 
columns and diagonally* 

Bow (R): Arrangement by row is shown by example U, for instance, in 

which each row has a different value of place (within the email square) • 
Columne (C)t An illustration of arrangement by columns are direction and 
also in example iu The values of these attributes differ in each column* 






r 



The arrangement in example 1; is, therefore, CCR (direction, size. 



iv; 



end place)* 
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Mlgonals (D or D® ) * Let x p y and z be vd. ues of the attribute* These 
can be arranged diagonally in either one of the following wayss 

D* x y z D«: x y s 



‘ i 
-t 

i 



i 







Ml. 



z x y y z x 

7 » x z x y 

Tho arrangement here is similar to that of a "latin square 11 and 
the diagonal may run from the uppor left hand corner to the lower right 
hand comer (D) or else from the upper right hand to the lower left hand 
corner (D®)* Any one of the values f, g, and h may be assigned to the 
positions in the above diagram that are indicated by x, y, and z* 

The following arrangements of D show how the four functions 




defined previously were arbitrarily arranged diagonally* (Note that 
the permutations a - a* and b - b* have been defined for the case 
where the values are arranged either in rows or in columns) * For 
the "latin square 11 arrangement these have been arbitrarily redefined 
so as to specify which value occupies the main diagonal* for a, this 
is the value occurring twice within the row or column (g); for a*, the 
value occulting once (f)j for b* it is one of the extreme values (f or fa)) 
and for b* # the middle value (g)* For D®, a similar arrangement of 
functions was used (with diagonals reversed)* 
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Teat Xtgaa 

Itena employed in the tests are of three classes: 

Class At Class A includes items in which the figures vary only in 
one or more of the three attributes: size* direction, or place* An 

item of this class can be defined by specifying the function and 
arrangement of each attribute varied* For instance, example 5 is 
defineable as: 

place: b D* 

direction: a* D 
site: b* R 

That is, place varies diagonally according to function b, direction 
by the other diagonal according to function a*, and sise by rows 
.according to function be* 

Items in which either two or three attributes vary were also 



included in the tests* The combinations of arrangements employed 



rendered the subtests described in Table 2-1* 



Table 2-1 



i s Snbtests of the Analytical Ability Teat - Claaa l* 





Threes 



CRD 



Two diagonals 



D*DR 
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Thus* In the RC subtest there are two attributes, and the figures 
are arranged by rows and columns only (see example 1), and in the D*BR 
subtest there are three attributes, and the figures are arranged by each 
of the two diagonals as well as by rows (see example *>)* 

The universe of items defined by the above specifications was 
sampled and 1*8 test items were constructed. 

In addition to the specifications outlined above, it was necessary 
to make certain arbitrary decisions, e.g., as to which sizes, directions 
and places to employ, whether f , g, g in items of function a should be 
arranged from top to bottom (right to left) or vice versa, and so on* 

Class Bi This class includes items where shape is an attribute which 
ia varied* For instance, in example 2 three different shapes are arranged 
by columns* Items of this class may vary also in other attributes accord** 
ing to one of the functions and arrangements described above: e*g*, example 2 
in addition to having shape as one of the attributes varied, has figures 
varying in place: be Bj end in site: b R* 

Class C» Solution of items of this class presuaably requires a greater 
degree of abstraction* The main feature of these iteais is that the 
function le exhibited either by meane of different attributes or by means 
of different sets of values of an attribute in the right and left hand 
columns, and the subject is explicitly Instructed to disregard arrangement 
by rows* In exampls 6 for instancs, in the left hand column, the function 
b ia exhibited by means of the attribute place) in the right head column 



f 



I 



-.■3.-d 






IERJC 






Section 2*1 (10) 



the function is also b, and is exhibited by direction* From these two 
columns the subject must find the rule, i*e», that function b must be 
employed on any attribute* The figures in the middle column vary 
neither toy place nor direction but by size* Saploying function b, the 
subject will understand that the correct answer is distractor No*8* 



Each column may exemplify more than one function as is the case 
in example ?• . In the left hand column the figures vary by size 
(function b*), by place (function b), and by direction (function be-)* 

In order to arrive at the correct choice in the middle column, the same 
functions must be employed, though not necessarily for the same attri- 
butes* It can be shown that No*l is the only appropriate answer 
amongst the alternatives presented* 



2*2* Pis tractor a of the Analytical Ability Test 



The usual method of constructing distractors calls for the 
creation of such alternatives as the subject is lilcely to choose; 
this is determined either by the intuition of the teat constructor 
or, empirically, by first presenting the test items to a group of 
subjects in an open-ended form* By contrast, the method employed 
in the present study relies on the facet design used in item con- 
struction, which allows for the systematic construction of distractors* 
Sines the choice of distractors must be assumed to determine to a 
large extent item difficulty as well as ths relationships of the items 
to out another, the construction of truly parallel forms of the test 
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if, not possible without employing this approach, which is based on 
facet design (see section 1.3). 

Another objective was that of obtaining distractors whose 
order of difficulty is predictable a priori , (to be verified, of 
course, by the test results) • This was achieved by manipulating sys» 
tematically the degree of s j jn i l a g ife between a given diatractor and 
the correct answer. 

Take, for example, an item in which three attributes are 
varied. The correct answer presents, by definition, the correct 
value on all three attributes. The distractors most similar to 
this answer have the correct value on two of the three attributes 
and an incorrect value on the third attribute. A less similar 
diatractor has a correct value on only one attribute. Finally, 
the diatractor least similar to the correct answer has incorrect 
values en aUthree attributes. Example k of the appendix, which 
is a Class A item, shows that there are eight distractors (including 
the correct answer) differing in respect to the attributes which 
have the correct value, as shown in Table 2-2. 

For the incorrect value of the attribute a value was chosen 
which: (a) appeared in one of the 8 squares of the test item, and 
(b) was closest to the correct value (e.g., in example It where the 
correct answer is figure which is large in size, the distractors 
which have incorrect values for this dimension are medium, not snail in 



20 . 



I 5 






: 'X 










Section 2.2 (3) 



Table 2*»2 





Siae 


Direction 


Place 


Distractor 

No. 


Correct answer 


+ 


+ V 


♦ 


7 


1 attribute wrong 




+ 


- 


2 


1 attribute wrong 


+ 


em 


+ 


1 


1 attribute wrong 


- 


+ 


+ 


k 


2 attributes wrong 


- 


* 


mm 


8 


2 attributes wrong 


mm 




* 


6 


2 attributes wrong 


+ 


m 


- 


5 


AM attributes wrong 


- 


m 


- 


3 



+ The dlstraetor is correct on this attribute 
- The dlstraetor is wrong on this attribute 



A dlstraetor constructed by the above rules could be identical 
with one of the 8 squares of a test item, c,g. dlstraetor No*2 in examplel* 
which is Identical with the middle square in the top row* This might 
entice the subject to choose such a dlstraetor, only fer this reason* 

Care was therefore taken to have ail items of the test# as far as possible, 

■ ii 

imolud* am equal ntnaber of dlstrectors identical with one of the squares. 

i ■■ - ...... . . ^ ' ....... ...... 

The number of distraetors for two-attributs items was also kept 
at eight# so as not to introduce an additional determinant of difficulty* 
This did not sake it possible to have a design of distraetors which was 
as "ti^ aa tha ona uytd for the three*>attribute items# but as far as 
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possible the same principles were observed. The design of these items 
is illustrated in Table 2-3 (by example 1 of the appendix) » 

Table 2-3 

Design of Distractors in Two-Attribute Items - Example 1 



Size 



Flace 



Correct answer 
1 attribute wrong 
1 attribute wrong 
1 attribute wrong 

1 attribute wrong 

2 attributes wrong 
2 attributes wrong 
2 attributes wrong 



+ 



+ 

+ 



Distractor 

No, 

3 

k 

1 

8 

$ 

1 

2 

6 



+ The distractor is correct on this attribute 
- The distractor is wrong on this attribute 



As to distractors of items belonging to Classes B and G* the 
reader can satisfy himself by inspection that they have been constructed 
in a similar systematic way, but space does not allow us to go into 



details. 
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2.3* Ai^ly tlcil-Abi liby Teat Booklets 



In the pilot study, after all items were drawn by our draftsman, 
it soon became/ clear that the degree of precision required for the 



drawings for this kind of test is very great* This tended to make the 
drawing of items very expensive; therefore a different procedure was 
adopted for the next version of the experimental test* Instead of 
drawing each individual item, standard figures were designed by our 
draftsman* There were six different figures, and each of these was 
drawn in four sizes, four directions, and four places* The resulting 
38 1* combinations were photographed and printed* Thus we had at our 
disposal 381* different kinds of labels, bound in small booklets* More- 
over, sheets were prepared on which the empty frame of an iter* appeared* 



Such materials may serve for the construction of an unlimited 
number of items; alllthat remains to be done is to paste the appropriate 
labefe in the comet squares, and then mimeograph the sheet* The work 
of the draftsman can thus be dispensed with and, though it involves an 
initi al l y larger expense, our procedure is actually more economical in 
the long run* Moreover, the use of these standard figures permits a 
far greater degree of precision than is usually attained otherwise* 



Another advantage of the procedures is that the differences between any 
given attribute remain constant between items* This el imina tes the 
possibility of the draftsman inadvertently determining the degree of 







difficulty of an item by introducing different degrees of similarity 



between values* 
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For the first phase of the study* four alternative forms of \>ne 
test were prepared* Each form contained thirty-two items and four 
practice items* The items were allocated to the different forms so that 
each possible combination of two items of Class A would appear in at least 
one of the forms* (Practical considerations did not permit implementing 
such an allocation for items of Classes B and C.) This allowd us sub- 
sequently to compute correlations between any two items* 

The four forms were designed to contain the following subtests; 
each subtest consisted of eight items* 

Form 1: DR* D«D, CCR, CRB of Class A* 

Form 2t RC* B»D, CRD* D ? DR of Class A* 

Form 3» RC* DR* CCR, D* DR of Class A* 

Form bt RC* DR of Class A; Class B; Class 0. 

In Class A* for the two-attribute subtests* eight combinations 
of attributes and their functions were chosen* and these remained the 
same in all three groups* Thus* an item having the following attributes 

* A 

and functions? sizes a*j places b* appeared in subtests RC, DR, and D«D. 
Similarly* items for the three-attribute subtssts wore equal in the attri- 
butes and their functions* 






r i 



i i 






For the second phase of the study* another form* Form 5* was -con- 
structed* tfee items for which were chosen on the basis of experience gained 
ill the first phase* Specifically* it wae deemed advisable to include 
items of eaoh one of the above groups* Further* additional items of t 
Clessi C end Class B, were constructed* Form ^th®n* included three items . 
each of RG* DR* D*D* CCR* CRD* end D*DR* six items of Class B and nine 
items of Class C (end three practice items)* 
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In Font 5# as the previous forms, the two-attribute items 
within Glass A had parallel items* The same was true of the three* 
attribute items in Class A* 



2*2l* Items of Mechanise! Comprehension Test 

One of the ma jor distinctions we have attempted to make is 
between "knowledge of physics 11 and "mechanical comprehension" • For 

success in a vocation involving handling tools and the like* knowledge 
of physics is certainly helpful* But practical know-how with instruments 
is more immediately relevant* and such know-how may exist "viscerally" 
without explicit technical grasp of any physical laws that may be involved* 
One may be able to drive a nail well with a hammer without being able to 
state correctly any laws about force* acceleration* and the like* 

Existing tests were examined from this point of view* and appear 
to contain a mixture of items* Some are on pure physics; others are 
on how to operate something mechanical - which could be answered correctly 
without any dear understanding of the physics involved* 

The more purely "mechanical comprehension" items were culled by 
the rule that they should deal with some real-life process like moving 
something* lifting* revolving* rolling* dropping* breakirg* ete** in which 
real-life instruments are involved and which are of common occurrence* 

This requirement ultimately defined one facet (Facet F, below) for item 
construction* This requirement is also practical for 8th-gradera* lor whom 
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it would not be very appropriate to test “knowledge of physics 11 (even if 
this were relevant for vocational guidance.) 

Exiting items fulfilling the “mechanical comprehension" requirement 
were looked at closely to see what further facets seemed implicit ii them. 
Afi earlier version of the faceted definition for this test was quoted in 
the research proposal* In the course of our work this definition was 
extended and sharpened so that it can now define any item (tf such a test 
and also be used to generate additional items* 

The revised definition runs as follows : 



(l* absolute) 

Which of two or more ( ) states of (element ) 

(2* relative) 

C - - D 



(X. lead to ) (!• absolute) 

H ill ( ) a relatively appropriate specified ( ) 

(2* result from) (2* relative) 



(1* absolute) 

state of (element Bj in a situation involving ( . ) (mechanical 

3 (2* relative) 

(is ) 

activity of type F*) where there j m intermediate consideration • 

* (is not) 
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List of elements in facets B and F 



B i 

(attributes of motion) 

1* length of path 

2* shape of path 

3. direction 

velocity 

3* acceleration 

6. duration 

7* amount of force 

8* distribution of 
force (torque), 



(geometric attributes) 

X • place 

2* shape 

3* volume 

k* circumference 

5. length, height (where 
volume is not relevant) 

6* angle 

7* relative position 
8* area 



F 

(type of activity) 

1* moving 

2* lifting/lowering 
3. revolving 
U* keeping in position 
5. rolling 
6* dropping 
7* breaking 

8. arresting movement 

9. starting movement 
10 . keeping path 



The definition can best be illustrated by examples taken from the 
test. For instance, let us contrast the definition of a question asking 
uhich arm of a rotating sprinl&er will be the first to arrive at a given 
point (see the example given in the appendix), with that of a question 
Peking whether a round o.- many-sided object will travel farther with the 
acceleration gained in traversing an inclined piano. Ihe first qusstion 
asks which position willrssult from a spocifiod shape (Bj.* 2.7J *-* 2| B j* 2 * 2 > 
while th%wcspdi WVd.re%,the^3^^ specified length of 

patb (Bit,2.2t th ® second question, ^ d iffere nt 

shapes aw.sgivenraii#!«l8q.^;(tpli^se^is^ a ** P aslwd 

about.ari ThersJtori^i^ Aothilaqqts AendD.the second element ("relative") 
applies. On the-pther hand,, in the sprinkler question, only one shape 
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is given and therefore “absolute” in facets A and D is the appropriate 
description# The sprinkler question involves the mechanical activity 
of revolving (F^) while the balls question deals with rolling (F^) # in 
the sprinkler question, in order to determine how the shape of the 
sprinkler leads to the specified position, the subject has to consider 

i 

in which direction the sprinkler will novo. There is therefore one 
intermediate consideration, that of direction. In the other question, 

j thB solution can b ® arriT9d »t directly without any intermediate considera- 

j tione. (Both questions deal in situations involving absolute mechanica l 

activity (%)} there are no questions involving relative mechanical 
activity in our test.) 

i ' 

t ■ 

The fruitfulness of the definition is attested to by the fact 
that it has resulted in the construction of items not previously included 

: , in mechanical comprehension tests. Examples are questions about the 

possible Influence of a curved accelerating path on the motion of an 
' / object after it leaves the path, about the relative efficiency, of banter* 

with handl8 " of different lengths (see appendix), and about the 

1 i . I . ■ * ' i ' -<• • ( 

I ?• screw threads on the direction the screw must be turned and the 

depth to which it can penetrate# 

,fi 

R «!#■*■, to ue, however, that 1* would be impossible to 

| | construct itmas for all atructuples (combinations of elements from all 

if facets) of the above definition because seme of them did not seem to 

U correspond to any empirical reality. Ms therefore proceeded to construct 

| | items of as many structuplea as was feasible and sampled from these 81 
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As shown in the examples given in the appendix, each item consists 
of a drawing, a question, and three alternative answers. The answer "same" 
diffbrincd V»d the like^ is the correct answer for very few items included 
in the test, as well as for one of the examples given at the beginning of 
the test* 



Items to be included in the test were divided into three groups 
of 27 each* Three alternative forms of the test were constructed, 
each of which included two of the three groups* Each form thus included 
5h items* This made it possible subsequently to compute correlation 
coefficients for any two of the 81 items* 

2*6* Collection of Data 

The three foxus of the mechanical comprehension test and Koxms 1 - i» 
of the analytical ability test were administered to a sample of 537 pupils 
which was constituted as follows* *** l - T,C! * v ‘ *‘ :J 



2*5* Mechanical-Comprehension Test Booklets 







lhO pupils 



256 pupils 
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Section 2 #6 (2) 

Hie vocational high school classes contained only boys; all 
other classes were mixed* A total of 1*17 boys and 220 girls were tested. 

The mechanical and- analytical ability tests were both given on the 
same day. Eaoh test required 1/2 - 3/1* of an hour. The alternative 
forms of each test were distributed amongst the students according to 
the order of seating, students sitting next to each other always getting 
two different forms# 

Form 5 of the analytical ability test was gLvwi at a later date 
to a sample of 511 adults of both sexes# 

The subjects wrote their answers on answer sheets. The answers 
were subsequently transferred to punch cards# 




. :* p 




2#7# Some Mew Methods of Analysis 
In order to analyse the structure of interrelationships between 

4 p if, ,! ’ * "l ' • • " . _ f . . t 

items and between groups of items in both the analytical ability and the 
meohanical con^rehension test, a new method of multivariate analysis was 
employed# A program of Smallest Spaos Analysis has been recently developed 

.1 V'»u ol-\' L J. i> Of _ V v,i It ■ ■. ‘jn ' .. ■ 

by Outtaan and Lingo** (tha 0-L 3Sfc-I). Thie la aeaantially a aynmetrio 

Cl.C^.0 f, £’,JLG if ft’ FftO Qf- </i ■■ . 

analysis resulting in a parsimonious Buelldean presentation idiioh has 
a monotone relation to the origineily given distances bstwesn veriablss 

(aaa rtfaxanoas 10, U, lit, 15). Iha ordar of thaaa dlataaeaB, tat not 

cmh cfc &oo b) i® feotss? t - r / x ■ fe 

their absolute slses, i» preserved, and tha program derives the smallest 

possible Euclidean apse* for thaae itm# 










Section 2*7 (2) 






For the purpose of our test, the distance function Dy used 
(where 1 and j are any two items), was defined as follows* 

a ai " i* subject s answers item 1 correctly 
(0, otherwise 

Then D^j- § .( a ei * *ej^ 

where B denotes the expected value over the indicated subscript# Expanding 

r* m 

D ij " Pi + Pj - 2Pij 

Khar* Pj_ ■ the proportion of subjects who answered item i correctly 

Pij " the proportion of subjects who answered both item i and 
item j correctly* 

Dij varies between 0 and X# The coefficient of similarity between 
two items may be defined ass 

C i j * 1 * D ij 

This coefficient also varies between 0 (perfect dissimilarity) 
and X (perfect similarity)# It has properties making it especially suitable 
for the analysis of test items# If %-j is used directly as a distance 
. function, a group of items which forms a perfect scale will fall on a 
straight line# If the items of the test can be described adequately by 
a three-dimensional space, then our coefficient of similarity tends to 
mCke the first principal axis of the space present essentially the 
order of difficulty (pi) of the items# Hence, the relationship between 
content of items (thb ^description of which is usually attempted by co- 
efficients such as Bsarsoi* x) is being described by the two-dimensional 
space of the remaining two axes# 
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Section 2.7 (3) 

In addition to printing out the coordinates on each of the 
prinolpal axes for eaeh itam, the SSJUX program also prints out the 
corresponding Shepard diagram and coefficient of alienation. The 
Shepard diagram is essentially a soattergram where each point represents 
the distance between two items; one axis represents t£e original co- 
efficient of similarity or distances (as defined by the above coefficient 
fry) and the other axis presents the distance in the n- dimensional 
space calculated by the program. The coefficient of alienation 
I refers to this relationship between distances and varies between 

0 and 1. 








3# FINDINGS 



3*1* Analytical Ability Teat 



3.1.1* Relative Difficulty of Subteata 



It was found that the facet structure of the items could predict 
the comparative difficulty of subtests, as measured by the mean number 



Class At In constructing the items it was assumed that items vaxying 
in two attributes would be more difficult than those vaxying in three* 
In addition, arrangements by diagonal were expected to create more 
difficulty than arrangements in either horizontal or vertical direction# 
These two predictions give xise to an hypothesis of partial order of 
difficulty, which is illustrated in Figure 3-1, where subtests are 
arranged from top to bottom in increasing order of difficulty* 



- Inspection of Figure 3-2 shows that the prediction is fully boms out* 



The bar-diagrams give the distribution of correct answers in the average 



of items (0 to 8) answered correctly* 





3-l » Hypothesized Relative Difficulty of Class A Items 
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Figure 3-2 Number of items answered correctly sn six subiests of class A. 








Section 3*1*1 (2) 



number of items answered correctly for different Glass A subtests* 

Data for two alternative forms for each group of items in this Glass 
are included* making it evident that the findings were replicated 
and can be regarded as reliable* 

As stated* only a partial order could be hypothesized* Thus* 
no predictions were made about the difficulty of CCR as compared to DR 
or that of CRD as compared to D*D. The relative difficulty of the 
members of these pairs of groups can be determined empirically* In 
our case* however* the differences were very small aid inconsistent 
in direction* Ibis suggests that CCR is roughly equal in difficulty 
to DR* and CRD to D'D* 



The order of difficulty shown in Figure 3-2 was replicated in 
Fom £> where all six subtests of Class A ware included (with three 
items in each subtest)* 

Class Bi Items in Class B in which shape was varied* were on the whole 
easier than Class A items* This was to be expected* since the sliapa 
of the figure is intuitively more salient than such attributes as its 
else or direction* and figures differing in shape appear more different 
from each other than figures differing* e*g** only in size* 

In Fom it* the average number of items answered correctly in 
Class B (4*3 items) was between that for group RC (4*9) and DR (3 *9)* 
Within Glass B* itms of varying dsgrees of difficulty are to be found* 
These items vary also in other attributes in addition to shape (see 
section 2*1) * 
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Section 3»1«1 (3) 



In Form 5, sane of the Class B items have arrangements by rows 
and columns only, as is the case with Class B items in Fora k* These 
were again - ntermediate in difficulty between subtests RC and DR of 
Class A* However, another kind of Class B items included in Fora £, in 
which two attributes were diagonally arranged, was much more difficult 
than these subtestsj the number of subjects which solved these items 
correctly was not much higher than that which solved correctly the 
most difficult items of Class A, D*DR, which also had two attributes 
diagonally arranged* 



Class Ci In Fora Class 0 items were the most difficult} the 
average number of items answered correctly was 3*2, as compared to 
lu9, U*3 and 3*9 for the other groups in this fora (see above)* 

Since the means tended to remain fairly constant between the different 
test forms, it can be said that Class C items with a mean of 3*2, are 
more difficult than groups CGR and DR and less difficult than CRD and 
D*D (see Figure 3-2)* 






In Form $ a new kind of Class C Item was introduced* In these, 
two or three attributes were varied in each column* These new items 
turned out to be very difficult, more so than Class B or the groups in 
Class A} ^ only one of these items was answered correctly by over 20# 
of the subjects and some were answered oorrietly by only * It 
appears that for most practical purposes these exceptionally difficult - 
items will not be useful since they possibly differentiate only a few 
extreme cases* The question as to the nature of the abilities required 
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for solving these items is of theoretical interest and still awaits 
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3*1*2* Scalability of Subtosta 



An interesting feature of the relationship between subtests 
becomes apparent by inspection of their joint distribution matrices* 
These show that for seme subtests it is not only the case that subtest 
x is more difficult than subtest y, but also that the correct solution 
of all or most items of subtest x, is a necessary condition for solving 
items of subtest y (with few exceptions)* Table 3-1 shows an example 
(subtests CRD and DR in Form 1)* 



Table 3 -1 

Joint Frequency Distribution for Numbers of 
Correct Responses (DR and CRD) 



As shown in the above table, there are only four cases (i*e* less 
than 2*1# of the total) who failed to solve four or more questions of 
subtest DR and who nevertheless solved four or more questions of subtest 
>* We can say that DR and CIS) form a kind of primitive scale * The 
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Section 3.1*2 (2) 

ability to understand items of DR is therefore in a certain sense a 
"prerequisite 11 for an understanding of items of CRD. It should be 
noted that although this result is a plausible one, it is by no means 
obvious. Compare, for instance, the joint distribution of answers for 
Classes B and C (in Fora k )* Here it will be seen that although Class C 

is more difficult than Class B, these subtests do not fora the kind of 
scale that was obtained for DR and CRD (see Table 3-2). 

Table 3-2 

Joint Frequency Distribution of Number of 
Correct Responses for Classes B and C 



\ Class C 
Class B\ 


0-1 


2-3 


ti-5 




Total 


0-1 


13 


3 


2 


1 


19 


2-3 ! 


'11 


10 


11 


3 


35 


U -5 j 


8 


15 


12 


6 


id 


6—8 


2 


20 


17 


12 


51 


Total 


31* 


1(8 


k 2 


■ I 


lii6 



Table 3-3 gives the joint frequency distribution of all pairs of 
subtests in Forms 1-1*. Since some of the subtests appeared in more than 
one fora, overlapping prevented us from including them in Table >3. 

These three subtests are presented in Table 3-3(a) given in the appendix. 
Explanation is still lacking as to what factors relate to the difference 
in scalability. At ary rate, the notion of subtests forming scales seems 

30. 
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Table 3-3. Joint Frequency Distributions cf Number of Correct Sesponees in the. Subteste of Forme 1-4 
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Section 3*1*2 (3) 



to be worthy of farther investigation and is currently being applied in 
another atucfy which deals with achievement tests* Achievement may 
progress by more sharply defined scales than analytical ability. 



3*1*3* Interrelations of Items 

The coefficient of similarity described above (section 2*7) * 

% 

was employed in the first analysis of the analytical ability test to 
represent the relationships between individual items* A three- 
dimensional configuration with a rather good fit was obtained by the 
SSA-I program* The following coefficients of alienation were found 
for the different formal Form Is *lU£, Fora 2* .116, Fora 3i .107, 

Fora hi *162* Fora $t *075* 

Using the coefficient of similarity* the first axis of the space 
essentially represents the degree of difficulty (see section 2.7). The 
second and third exes were studied in order to find out how the facet 
structure of items relates to their configuration, apart from difficulty. 
Specifically, there are two kinds of relationships which could be expected: 

(1) Items which are similar to each other in n facets may be 
expected to be closer to each other in the two-dimensional space than 
items similar only in a proper subset of these facets* This is an 
application of what has been called the principle of contiguity (see 
references 2|, 11) and leads to the prediction of structures such as the 
simplex and the jtrcumplex (see references U, 6). 

kO* 



Section 3.1*3 (2) 



(2) The two-dimensional space can be partitioned into contiguous 
regions each of which contains (with few exceptions) those and only 
those items which are similar in one or more facets* Roughly spea kin g 
this may be interpreted as showing that items of one region are on the 
whole closer to each other than to those of another region, though 
this does not rule out the possibility that some items belonging to 
one region are oloser to those of another region than to the remaining 
items of their own region* 

While the correlation coefficient pertains to individual variables 
or items, the concept of contiguous regions pertains to sets of variables 
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or items, and, being based on the correlation concept, is a concept of a 
higher order* The empirical use of the concept of contiguous regions 
has been facilitated recently through the availability of new methods 
of computer analysis such as those described in section 2*7* 

Investigation of the two-space showed that our test results did 
not conform to the expected relationship stated first* Furthermore, 
parts of the configuration of these sets were not stable from form to 
form (i*e*, items relatively close to each other in the two-dimensional 
space in one of the forms might be relatively far apart in the other, 
and vice verea*) 



What is the explanation of this finding* The rather large 
number of subjecte filling in each form (lh«> or more for each one of 
the Forms 1*4 and £ll for Form $) argues against the possibility of 
the differences being caused by sampling error alone* The retest 
reliability of each form individually is also evidenced by the fact that 
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Section 3*1*3 (3) 



ita many items can be represented in a three-dimensional space 5 there 
is a definite structure to each form separately, and structure provides 
a lower bound to retest reliability (see references 1 , 3 , 5 ). Further 
evidence in this direction comes from indications of external validity* 
As reported in section 3.2.i», correlations of subtests of the analytical 
test with scores on the mechanical comprehension test ranged from *33 to 
•58 (with one half of the computed correlations coefficients being .50 
or more) » 



There remains, then, the explanation that the different configura- 
tions in alternative forms are due to the differences in the sequence of 
items appearing therein* 



In spite of areas of instability, contiguous regions could be 
found - replicated over forms - which contained items having some facet 
elements in common* 



Within Class A, arrangements of items tended to result in groups 



of items j forming a region, but in most cases such regions were partly 
overlapping rather than contiguous* 



/ 



Furthermore, the function employed in the construction of the items 
had an effect on their configuration, A region containing items in which 



function a or a* predominated and another region containing those in which 
functions b or b* predominated could be discerned in all alternative forms* 
The difference between a and a# and that between b and b* (i*e*, the 
different permutation of values) did not seem to affect the correlational 
structure* To describe this finding more precisely let us introduce the 
terms a-items and b-it tms which are defined as follows s An a-item is 
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one in which either all attributes or two out of three attributes 
vary according to function a or a** A b-item is defined* mutatis 
mutandis* in the same manner* Now it can be stated that in all 
forms there were two contiguous regions one of which contained all 
a-iteras and one of which contained all b-itema (with very few exceptions)* 

It will be noted that every item varying in three attributes 
can be classified as being either an a-item or a b-item. So can 
two-attribute items in which both functions are either "a-f unctions" 



(containing only a or a*) or "b-functiona"* Not so those two-attribute 
items which contain one "a-function" and one "b-f unction"; here 

neither a nor b predominates* It was fouhd, however* that in the 
DR subteot the region into which the item fell depended largely on 
whether the attribute arranged diagonally varied according to an 
"a-function" or a "b-f unction"# 
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In order to find out how two-attribute items of the RC sub- 
test fitted to the overall configuration* one of the investigators 
independently judged which of the attributes of these items (size* 

4 ’ * 

direction or place) was salient * It was argued that* e#g#* when the 

* ' ■ « J , 

direction of the figure appeared intuitively to be the most salient 
attribute* the subject would* on first looking at the item* pay 
attention to direction and txy to solve this item accordingly by 
choosing a distractor which would exhibit the value of direction 
seemingly most appropriate to him* It was then found that those 
items of RC in which the salient attribute varied by itanction a or a* 
fell* almost invariably within the region of a-items whereas those in 
which the salient attribute varied by function b or be fell within the 
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region of b-i terns* This pattern was not found for D*D items* 



In conclusion* it can be stated that the function facet exerts 

* « 

an influence on the structure of intercorrelations. As the foregoing 
will have made clear, however, only certain aspects of this structure 
can so far be explained by the facet design: the final structure of 

the correlation matrix stilleludes our analysis* Apparently the 
difficulty of items plays a very large role in this structure* In 
the present test, where items formed a rather homogeneous set, the 
second and third dimensions which represent the similarity between 
them when difficulties are held constant (see section 2.7), account 
for only a minor part of the variance. It seems, then, that similarity 
between items is so great that those differences between them which are 
defined by the facet structure will have a small impact compared to 
that of other factors* In other words, tapping a wide variety of 
abilities should be expected to yield a more stable structure than the 
more homogeneous group which constitutes the present test. The struc- 
ture of the latter may be susceptible to subtle influences not picked 
up by mr facet definition* While it is in the nature of the case 



that these influences cannot be identified with certainty, there 
appeared to be indications of perceptual factors playing a role* This 
notion will be discussed in section 3*1*6* 




3*1J|* Interrelations of Sub-bests 



Analysis of the intercorrelations between individual items has 
shown that the resulting structure can be interpreted to a cer tain 
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extent in terms of the functions specified by the facet design, whereas 
other facets did not seem to be represented in the structure (section 
3»1»3). An additional analysis was carried out to determine whether 
correlations between total scores obtained on various subtests are 
such as to render a structure which is interpre table in toms of the 
facets which serve to define these subtests, namely, arrangement of 
values and class of item (see section 2.1). Data obtained for Form £ 
were used in this analysis, because this form contained the greatest 
variety of subtests s there were 12 subtests of two or three items each. 

Product-moment correlations were computed for all pairs of total 
scores of these 12 subtests. There were three subtests containing 
Class C items which differed in the number of attributes varied in each 
column (see section 2.1). It was found that these had almost invariably 
close-to-zero correlations with all other subtests. The three Class C 
subtests were not even strongly related to each other, the correlation 
coefficients being .20, .08, and .03. 

The intercorrelation matrix for Class A and Class B subtests was 
submitted to SSA-I. The resulting two-dimensional space had a good fit 
(coefficient of alienation .106) and is easily interpretable. As shown 
in Figure 3—3* the two-space can be partitioned in two directions. The 
boundary of one partition nitos more or less horizontally across the figure, 
dividing the three Class B subtests (i.e., subtests in which shape is an 
attribute varied) from Class A subtests. The other (close to ver t! cal ) 
partition, determines the number of diagonal arrangements occurring in 
thus items of the subtests. The region to the left of the figure includes 
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subtests X and 2 of Class B and two Class A subtests that have no 
diagonal arrangement! the middle region includes two Class A subtests 
with one diagonal arrangement! and the region to the right includes one 
Class B subtests and two Class A subtests with two diagonal arrangements. 




This simplicial structure of diagonal arrangements was replicated 
in an analysis of the Glass A subtests in Forms 1-3* Each of these 
forms included four of these subtests (with 8 items in each subtest)* 

It was found that "no-diagonal 11 subtests correlated more with "one-diagonal 11 
subtests than with "two-diagonal" subtests, and that, in fact, there was 
little overlap between these groups of correlation coefficients* These 
results are shown in Table 3-U# 
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Table 3-lt 

Correlation Coefficients between Subtests of Forms 1*>3 
having Two* One* or No Diagonally Arranged Variables* 



fvpf 




v. No diagonal 


One diagonal 
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It may be concluded that bjoth arrangement of items and their class 
affect the structure of interrelationships. It is noteworthy, though, 
that the structure is affected only by diagonal arrangements, whereas the 
difference between arrangement by rows and arrangement by columns does not 
seem to play any part. Again, this was replicated in the analysis of' 
Forms 1-lw Furthermore, neither in Form $ nor in Forms 1-U was there 
any indication of an effect of the number of attributes varied on the 
correlational structure. 



3.1«5>. Relative Attraction of Distractors 

Detractors for the analytical ability test were designed in a 
systematic manner as has been described above (section 2.2). This 
design made it possible to predict their differential attractions. 

In particular, distractors differing from the correct answer in one 
attribute were chosen more often than distractors differing in two 
attributes, and the latter type of distractors would be preferred over 
those differing from the correct answer in three attributes. In Table 3-5, 

the average number of subjects who chose distractors which were incorrect 

* , - 

in one, two or three attributes is given* The average was computed by 
dividing the total number of times a particular type of error was made 
(i.e., errors in one, two or three attributes) by the total number of 
times a dis tractor with one, two or three attributes incorrect appeared 
in the subtest. The data were averaged separately for the different 
subtests of Class A - CR, DR, etc. - and were drawn from Foms 1, 2,- 3, 
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Table 3-5 

Average Number of Subjects Choosing a Distractor Incorrect 
in One. Two, or Three Attributes, in the Different Subtests 



Number of 


Subtest 


attributes 

incorrect 


EC 


DR 


D'D 


OCR 


CRD 


D»DR 


One 


13.0 Uu2 n.S 


18.1 


22.5 


23.1* 


Two 


k.6 


5.2 


9.2 


5.7 


12.0 


7.7 


Three 


(not applicable) 


1.8 


3.3 


6.3 



The systematically manipulated similarity between the distractor 
and the correct answer is, then, a very powerful factor of its attraction. 
There were notable exceptions to this general rule, however. For some 
items the wrong answers would tend to involve choice of a distractor 
differing in two attributes from the correct answer, $nd these were 
preferred to distractors more similar to it (i.e., differing from the 
correct answer in one attribute only). Examination of these items 
suggests a reason for this. In several cases in which distractors 
with two incorrect attributes were preferred, these distractors were 
similar in all respects to the two figures of the matrix adj oining it* in 
either horizontal or vertical direction. Apparently, those respondents 
who gave these answers tended to look for a distractor in allrespects 

.Ut *. t thy w,| i&sv va 

similar to the row and the column of the mctrix.’ They made what will 
be termed adjacency-errors . 
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An adjacency-error refers to cases in which one of the dis tractors 
was exactly alike* or vsiy similar to,*** the other forma in either the middle 
row or the middle column* or both* if the two pairs in the horizontal and 
vertical are both alike (e*g*, distractor No .6 in example 3 of the appendix 
and distractor No*6 in example 5)* 



Table 3*6 shows that there was a tendency on the part of the 
subjects to make adjacency-errors; the average number of students who 
chose the adjacency-error is compared with the average number choosing 
other distractors* (Thus* if there was only one adjacency-error in a 
particular subtest* the number given is the absolute number of subjects 
choosing the adjacency-error distractors) • The comparison is made for 

the subtests of Class A within Forms 1 and 3# All adjacency-errors in 
one subtest of eight items were considered as a whole* The average 
was computed by dividing the total number of times an adjacency-error 
was chosen* over the numoer of possible adjacency-error distractors* 



1 



At. distractor is ' similar* when the two squares in either the 
middle row or the middle column* though having the same values 
for size and direction* have slightly different values for the 
attribute place* though neither of these values can be extreme 
(i*e«* not in one of the comers of the small square)* In Z 
addition for a distractor to be 'similar 1 * there must be only 
one distractor which is exactly like one of the two squares 
in the middle row or column* and no distractor similar to the 
other square* 
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Table 3-6 

Compariso n between Average Number of Students Choosing 



Subtest in 
Fom 1 


Subtest ija 
Form 3 


Average Choosing 
Adj acency-Error 


Average Choosing 
Other Detractors 




RC 


1i5.o 


8,0 




DR 


78.0 


9.9 


DR 




87.0 


10.3 


B«D 




U3.0 


13.3 




OCR 


1(2.8 


6*3 


COR 




68.5 


5.8 


CRD 




1(3.0 


1 luk 




D*DR 


35.0 


13.9 



To show how the availability of adjacency-errors interacts with 
similarity in influencing the subject 1 * choice, a further breakdown of 
answers was made i$$ Table 3-7 which compares the average number of 
subjects choosing a distract or incorrect in one, two or three attri- 
butes aliens (a) there is no available adjacency-error) (b) the 
adjacency— error loads to choice of a distractor incorrect in one 
attribute) (c) the adjacency-error leads to choice of a distractor 
incorrect in two attributes, (It so happened that there was no 
-adjacency-error possible in a distractor with three attributes in- 
correct.,) The averages were computed as in Table 3-6. 
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Table 3-7 

/ 

Average Number of Subjects Choosing Incorrect 





• 


I Available Adjacency- 
I Error Incorrect in 
I 1 Attribute 


• No Available 
Adjacency-Error 


ajrjru sr 

1 

Available Adjacency- 
Error Incorrect in 
2 Attributes 


Subtest 

in 

Form 1 


Subtest 

in 

Form 3 


I detractor actually 
1 chosen incorrect in* 


J&sbractar actually 
chosen incorrect in : 


Distrector actually 
chosen incorrect in* 


A 

attr. 


Z 

attrs. 


3 

attrs. 


1 

attr. 


2 

attrs. 


-3 

attrs. 


: 1 
attr. 


attrs. 


3 

attrs. 




RC 


17.0 

It 


3.3 


- 


10.2 


6.7 


warn 




ee 


DR 


[ 


23.8 

|[ 


3.7 


- 


li|.2 


6.3 


mm 


t. 


mm 






DR 


2lt.8 

' ' 
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15.2 


!t.l 
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mm 
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20.lt 
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1.0 


13.9 


lt.3 


2#0 


7.0 


13.3 


3.0 


CCR 
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28.3 


3.3 


1*3 


9.9 


lt.9 


2.5 


8*0 


2l|.0 


1.0 


CRD 
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mm 
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22.5 


10.0 


3.7 


2U.0 


17.3 
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22.6 


7.9 


6.2 


23.2 
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3.3 



The finding that its similarity to the correct answer affects the 
attraction of a extractor has important practical implications, since it 
now seems feasible to assign differential scores for each item by taking 
into account the type of distractor chosen. It appears, however, that 
possible adjacency-errors haws to be taken into account as well. * 
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3*1*6* Perceptual vs* Logical Processes 

Items of our teat were built according to a rule of construction 
the discovery of which was one of the tasks assigned to the subject 
(section 2*1) * The question may be raised as to whether the subject 
actually tried to elicit the rule underlying the item* or whether he 
attempted to find the solution by means of a different process* 



The effect of the facet design of items on the correlation 
structure should be expected to be maximal when subjects actually 
think in terms of these facets, i*e*, when they solve items by way of 
the construction rule* Such a process of solving items will be termed 
logical process* Inspection of Raven's Progressive Matrices - which 
served as a starting point for work on our pictorial analytical ability 
test • suggests that other processes may be at work in this tost* Some 
of Raven's distractors seem to be quite unlikely a choice because they 
contained figural elements not included in, or very dissimilar to, thosp 
in the matrix* Conversely, one or two distractors of an item may 
immediately suggest themselves as possible correct , solutions because 
of their similarity to the figures in the matrix. When the subject is 
influenced by considerations such as these in the solution of items, 
we shall say that perceptual processes are at work* While it seems 
reasonable to assume that such processes are operative in the solution 
of Raven's matrices, there is no formal evidence to this effect* At any 
rata, in constructing our test, it seemed desirable to reduce the operation 
of perceptual processes to a minimum so as to maximise the effectiveness 
of the facet structure* 
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It soon became apparent that by constructing the dietractors of 

% 

our test in a systematic fashion (see section 2*2) there were fewer 
possibilities for perceptual processes to take place* Thus, intuitively, 
none of our distractors appear to be as unlikely a choice as seme of Haven's 
di street ora* 

However, the systematic construction of distractors by no means 
ensures that perceptual processes are ruled out completely* In the 
first experimental version which was employed in our pilot study, the 
number of figures served asan attribute and this apparently favored 
the operation of perceptual processes (see discussion in section 2*1)* 

This impression was reinforced by a small informal experiment, which 
indicated that in this version, judged similarity of the correct answer 
to the figures adjacent to the missing figure was negatively related to 
the degree of difficulty of the itmn* In this version, the missing 
figure was in the third row third column, i*e*^ in the lower right hand 
corner* In order to reduoe this affect in the revised version, it 
was thought advisable to omit instead the figure of row 2 column 2, i*e* 
the middle figure* Further, the revised version did not employ number 
as an attribute, nor was the Amotion f,g,h(f,g) included (see section 2*1)* 
It remained to be seen how successful these steps were in reducing the 
operation of perceptual factors* 

""v 

Tbs first indication of the operation of perceptual factors in 
the revised version (which is the one on whleh ell the analyses reported 
in the previous sections were carried out) comes from one of the findings 
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pertaining to subjects 1 preferences among the different distractors 
of an item* As stated (section 3*1*5), when the distractor which was 
most similar to the correct answer was less preferred* the more preferred 
distractor frequently was similar to the figures adjacent to the missing 
square* Wherever this occurs, it seems that the subject did not attempt 

a logical analysis of the test item, but rather used perceptual processes 
instead* 



Further analysis of subjects 1 errors substantiates this claim* 
It was attempted to find out whether making an adjacencgr~error was a 
“trait 1 * of a certain sample of subjects; i.e., whether the subject 
whose answer to one item is determined by adjacency willtend to make the 



same type of error in other items as well* It was found that this was 
indeed the case* Four of the forms were examined with regard to the 



tendency to make an adjacency-error* Within each form, all Class A 
items in which such an error was possible were comparod* In Form 1, 
there were 7 such items; in Fora 3, 9 items; in Form k , 5 items, and 
in Fora % 9 items, giving a total of 98 comparisons* In every single 



case the tendency to make an adjacency-error on one question in the 
pair correlated with the tendency to make a eimilar error on the other 

: ’ ... • V' • * 

question* The fact that the correlation was always replicated is 

, .■ * .... _ . - ' ■ 

evidence for the reliability of the finding* These results are shown 
in Table 3-8(a-d). 



i5s a farther teat of the operation of perceptual processes, 
the configuration of items ms examined, to determine whather those 
items in itoich the same attribute is salient would tend to fall within 
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a contiguous region* It was felt that analysis of subjects* incorrect 
answers would disclose which attribute of the test figures, in each 
instance, was the most salient* If the distractor chosen by the subject 
was identical to the correct response in one attribute, it could reasonably 
be hypothesized that the subject was attentive to, and based his choice 
of response upon, this attribute alone* The use of one attribute and 
disregard of the others was taken to indicate the saliency of the particular 
attribute in the specific item* 

This test was applied to all incorrect responses to each Class & 
item in Forms 1, 2, 3 and 5* It was found, through inspection of the two- 
space, that items with direction as their salient attribute indeed tended 
to fall into one region, but there were not enough data, nor were the data 
sufficiently consistent, to permit any conclusions on this matter* 

. It may be concluded, therefore, that perceptual processes are 
often resorted to by subjects taking the test; this is evidenced by the 
correlational structure as welli as by the subjects* preference for the 
specific distractors* To the extent that such perceptual processes are 
not taken into account in the facet design, they might be expected to 
interfere with the predicted correlational structure (see section 3ol*3)* 
Hence, the operation of such factors should be taken into consideration 
when constructing the test* This applies largely to tests employing 
figures as test material* In verbal tests, perceptual processes obviously 
should not be expected to operate, but it still remains to be seen whether 
thebe tests are subject to a different kind of interfering factors* 



3*2. Mechanical Compre hension ^eet 

3*2.1. Relative Difficulty of Items 

The first analysis of our results was intended to determine to 
what extent the facet structure of items can predict their degree of 
difficulty* Five elements from two of the facets appear to be related 
to degree of difficulty's Fs 3, B: 1*1, B: 1.2, B: 1.7, Bt 2.6« 

They are respectively: mechanical activity of revolving (difficult), 
length of path (moderately difficult), shape of path (easy), amount 
of force (difficult), and angle (easy). But the strongest relation- 
ship seems to exist between facet 0 and degree of difficulty* The 
median percentage of subjects who answered correctly items which can 
be solved without an "intermediate consideration 11 was 69 % whereas the 
median proportion of those items requiring “immediate consideration 11 



The relationship between facet structure and difficulty 
indicates that our facets defined psychologically relevant aspects 
of the items. 

3.2.2. Interrelations, ofiltensr v • ;• ^ 

Analysis by the G-L SSA-I program resulted in a three-dimensional 
space with a reasonably good fit (coefficients of alienation 0.11:57, 
0.2637# 0.1521: for Forms \ 2 and 3 respectively). As stated above 
(section 2.7)# the first axis of this space relates to the degree of 
difficulty of the items. When the space of the remaining two axes was 
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studied, the resultant configuration was not found to be interpretable 
in accordance with our facet design* Further, as was the case with 
the analytical ability test, the configuration was unstable in part, 
since it differed among the alternative forms of the test* This may 
possibly be explained as the result of the different sequences of 
items employed in the different foras* 

Even when this factor was taken into account, however, and 
only those subgroups of items which showed a stable pattern from one 
fora to another were examined, the item configuration of the two- 
space -itill could not be attributed to the facet profiles* It must 
therefore be concluded that various facets of the test — if at all 
bohavioraHy relevant — interact in a complex fashion, and that this 
does not permit prediction of correlation structure, at least until 
very extensive further investigations are carried out with a great 
number of items employing our facet design* 



3*2*3* Subtesta 

While it is not yet possible to predict relationships between 
mechanical comprehension items on the basis of our facet design, a more 
modest analysis of the configuration yielded by the SSA-I program 
seems to give good results* Certain subgroups of items were found, 
each of which satisfied the following conditions; 
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Section 3#2.3 (2) 



(•) The group of itons pertained to i certain sub-area of mechanical 
comprehension! (there wars questions involving free movement! the 
results of movement on the body being moved! the influence of 
length of radius on the resultant movement)! and 

(b) the group formed a quasi-scale; 

(c) which was stable over alternative foiras. 

Table 3-9 lists these groups. The numbers in each group are 
the item, numbers in the form of the test for which the coefficient of 
reproducibility was computed. The order of items in this table is 
from the most difficult to the easiest. Coefficients of reproducibility 
are given for each group. 

Table 3-9 

Subtests of Mechanical Comprehension and 
their Coefficients of Reproducibility 











.Coefficient 


Content 


Form 


Items No. 




of Repro- 
ducibility 


^ree movement 


1 


21, 27, 1*1*, 


18 


.91* 


Free movement 


1 


1(0, 19 , 17 




.88 


a* 

Free movement 


2 


23, 51*, 1*8, 


19 


.91* 


Free movement 


2 


38, 31*, 39 




.91 


Free movement 


2 


26, 17, 21* 




.89 


Movement and its effects 


1 


1*2, 38, 15 




.91* 


Movement and its effects 


2 


21, 29, 52 




.91* 


Length of radius 


2 


10, 56, U, U* 


.89 


Length of radius 


2 


51, 1*2, 27 




.91 
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Section 3.2.3 (3) 

The immediate practical advantage of this grouping lies in that 
a new version of the test (which yields several sub-scores), can now 
be constructed. These sub-scores are meaningful in that score profiles 
can be reconstructed from.th^gu Further investigation with the sub-scores 
can be expected to reveal more important theoretical relationships with 
outside variables than do total scores on mechanical comprehension tests. 

In fact, such a revised version of our mechanical comprehension test is 
now being prepared and will shortly be in use at vocational g uid ance 
centers throughout the country. 



3.2.1*. Relationship between Mechanical Comprehension and Analytical Ab-i n ty 
Correlations were computed between the total score of the mechanical 
canprehension test and subtests of the analytical test, for Forms 1, 2 and 
3 of the latter. (Subjects taking these forms took Forms 1, 2 and 3 respec- 
tively of the mechanical test. The subjects taking Form k of the analytical 
teat were equally divided between Forms 1 and 3 of the mechanical test - 
see section 2.6 - and therefore, the samples were too small for the com- 
putation of reliable correlation coefficients). Table 3-10 shows these 
correlations. 

Table 3-10 

Product-Moment Correlations between Total Scores of the 
Mechanical Comprehension Teat and Subtests of Analytical 

Ability, (Class A Items) 







Analytical Ability Subtesta 






Rfl 


DR 


DID 




CRD 




Form 1 


- 


.38 


.38 


.111 


.33 


• 


Form 2 


.58 


- . 


.50 


- 


.55 


.1*6 


Form 3 


•55 


.58 


m 


.51* 


«a» 


.1*1 
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Section 3#2.h (2) 



It will be seen from the table that the highest correlations were 
those between mechanical comprehension and (with one exception in Form 1) 
subtests RC and DR, (i.e., the easy subtests, which include two-attiibute 
items with at most one diagonally arranged function) while the lowest 
correlations were those with subtest D«DR (i.e*, the most difficult sub- 



test$ see section 3*1*1) • 

It should be pointed out that the size of these correlation 
coefficients bear evidence to vhe external validity of both tests con- 
cerned, and therefore also to their reliability • 

More detailed analyses of the relationship between mechanical 



comprehension and analytical ability are now being carried out on the 

t 

basis of subtests of the mechanical comprehension test* 

To obtain additional information on the relationship between 
mechanical comprehension and different kinds of analytical ability, a 
reanalysis was undertaken of the results of a study by Xraak (see 
reference 13). Kraak published the correlation matrix for ten sub- 
tests of mechanical comprehension, a verbal analogy test and a test of 
numerical progressions* 



Jki SSA-I space shows that the mechanical tests closest to the 
verbal analysis test, are two subtests which show diagrams or pictures 
of machines and in which the subject is called upon to explain their 
operation. However, another subtest showing such a diagram, is far removed 




from the verbal analogy test* 
level*, two arms of which are 



(This subtest shows aiftoee-axmed T-formed 
by springs, and the subject is asked how 
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Section 3 *2.1* (3) 

« 

a weight on one of the anna woul<J. influence the lew.) Four teata 
of apatial perception fora contiguoua regions of the two-dimensional 
apace resulting from the SSA-I. Theae are closest to the numerical 



progression test. 








Ju CONCLUSIONS 



In the present study a first attempt has been made to implement 
the facet theory approach in test construction and data analysis. 

The experience gained here makes it possible to evaluate the fruitful- 
ness of this approach, and the promise held out by it for further 
progress in mental testing* 

The first point to be made here is in regard to the possibilities 
of systematic item construction through facet design* Two kinds of 
tests were devised in this study* an analytical ability test aid a 
mechanical comprehension test* While each employed a facet definition, 
it appeared that the two differed in the extent to which use could be 
made of their respective definitions* In the analytical test, the 
facet design could be used to specify the items down to every detail* 

This was not so in the mechanical comprehension test, where the designs 
could be used only to determine the kind of relationships between 
attributes (such as force, speed, and Tolume) which the items pertained 
to* It was not possible to build items for each one of the profiles 
which could be generated from the facet definition of the mechanical 
test, whereas no such limitation was found in regard to the caalytleal 

celled for the finding of analogies and employed arbitrarily 
cheaan geomatrle figures* 

could be systeaa- 

tioally designed by naans of fleet* ’ ^ section 2*2)* Analysis of the 
: results tbowed that we were successful In constructing distrectors which 
diffdred iyst^ ^attraction*, opening up new possibilltile 



Section k ( 2 ) 



j for increasing the discriminatory power even of short tests (section 3 # l.J>}* 

j This advantage was not shared by the mechanical test, where it proved 

I 

t difficult to design distractors by facets* 



Considering the nature of the tests concerned, all this is just 
what one would expect* It only goes to show that tests may differ widely 
in the east with which facet design may be applicable to their construction* 
On the other hand, it seems safe to say that whatever the naturo of the 
test, facet design can be employed in item construction at least to some 
extent; the experience with the particularly unwieldly mechanical ability 
Questions seems to be evidence for this* 



u Tb8t th * fB08t8 employed in th« construction of the present test 

* 8re Psychologically relevant is shorn, first of all, by the fact that 
j th *y could P**dict the relative degree of difficulty of test items. This 

E ! 18 truB for analytical ability teat and, to a certain extant, for the 




mechanical comprehension test* 



Another question - which also concerns the psychological relevance 
of our facets - is that of the obtained relationship between the facet 
structure of the test and the statistical structure of its results* Here 



*» . ‘ .. i * ■ , 

•(•in, large differences ware found batman the too kind* of teats. For 
r*Mn>* itiloh have bean dlscuaaad, the data from the meohanical teat did 
not yield an interoorrolatioo matrix vhioh me interpreters in terms at 
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Section It (3) 




analysis (sections 3*1.3# 3*1*10 • Even here though, the success achieved 
was only partial, and many details of the statistical structure did not 
prove amenable to explanation through the facet profiles of the items* 

In considering possible reasons for this, a factor which acted 
as "noise" was hypothesised* the operation of what has been termed 
"perceptual processes" (section 3*1*6) • Supplementary analysis of the 

data lent support to this explanation* This result might prove to be 
important for future work on the construction of analytical ability 
test of the kind developed in this project, where, it is suggested, this 
factor should be taken into account* 




In sowing up the lessons of this project, we might say that this 
first attempt of carrying out all steps of test development systematically 
by means of facet design sterna to show the fruitfulness of the approach 
token* The degree of success achieved was uneven, differing with the 
kind of test and the nature of analysis conducted* At times it was 
possible to reveal the reasons for the difficulties encountered, and 
thus valuable experience has been gained for future work on test develop- 
rnent based on facet theory* 
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From which outlet will the water reach the tree first, 

1* From outlet A («) 

2» From outlet B (nj 

3* It cannot be established 



With which hammer will it be easier to pull out the nail? 

1* With hammer A(tt) 

2* With hammer B(aj 
3* It makes m difference 



Which ball will roll in an arc? 

1* Ball. A(«) 

2* Ball B(a) 

3* Neither of them 



Example of Mechanical Comprehension Test Item 



i 



Table 3-3(a)-(Cont.1 



RC^'\ 


0-1 


2-3 


lt-5 


6-8 


Total 


0-1 


12 


3 






IS 


2-3 


29 


b 


1 


•» 


3b 


b-5 


20 


7 


k 


1 • 


32 


6-8 


27 


26 


16 


6 


75 


Total 


88 


bo 


21 


7 


156 



Form U 



— i 

DR^^j 


0-1 


2-3 


. k-5 


6-8 


Total 


0-1 


13 


7 


b 


3 


27 


2-3 


5 


11 


9 


8 


33 


b-5 


3 


7 


12 


30 


52 


6—8 


- 


- 


b 


30 


3b 


Total 


21 


25 


25 


71 
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